Deep Semantic Segmentation of Built Heritage Point Clouds

L'abstract è presente nell'allegato / the abstract is in the attachmen

Fine-tuning and data augmentation techniques for semantic segmentation of heritage point clouds

This topic of this contribution falls within the broader debate on Digital Humanities. Experiencing and testing an approach that combines geomatics and its production of three-dimensional data of the built cultural heritage (CH) with information technology is the core point. In the digital CH domain, the ever-increasing availability of three-dimensional data, provides the opportunity to rapidly generate detailed 3D scenes to support restoration and conservation activities of built heritage. Concurrently, the recent research trends in geomatics are facing the issue of managing these heritage data to enrich the geometrical representation of the asset, creating a complete informative data collector. HBIM (Historic Building Information Modeling) constitutes a reference, and they typically rely on point clouds to perform the scan-to-BIM processes. These processes are still mostly manually carried out by domain experts, making the workflow very time-consuming, not fully exploiting the potential of point clouds and wasting an uncountable amount of data. In fact, parametric objects can be described through a few relevant points or contours. The use of Artificial Intelligence algorithms, in particular Deep Learning (DL) techniques, for the automatic recognition of architectural elements from point clouds can therefore provide valuable support through the semantic segmentation task. A proposal to tackle this framework was outlined in previous works, and the methodology here proposed constitutes a development of their results. Starting from those former tests obtained with the Dynamic Graph Convolutional Neural Network (DGCNN), close attention is paid to: i) transfer learning techniques, ii) the combination with external classifiers, such as Random Forest (RF), iii) the evaluation of data augmentation techniques on a domain-specific dataset (ArCH dataset). Besides, an investigation on how to make the whole workflow more functional and "friendly" for external users is carried out too. With regard to transfer learning techniques, the fine-tuning approach is proposed to understand if, also in the CH domain, it can lead to performances improvement, introducing a new scene in a pre-trained network. In fact, the peculiarities of each scene do not guarantee certain and definite results, as for other domains. This section is divided into two subsections: a classic fine-tuning and a fine-tuning with the addition of the RF in the final part of the prediction. In the latter case, the choice of adding the RF is due to the results obtained in some stateof-the-art works, where this classifier provides excellent results in a short time and even in the presence of relatively limited data. In this hybrid approach, the network weights are employed as well as in the classic fine-tuning technique. Then, the final part of the DGCNN performing the segmentation of the points is excluded, leading the network to be used as a feature extractor method; afterwards, a scene of the dataset never seen by the network is chosen and divided into one part for training and one for the test. Finally, the features of both parts are extracted, using the feature extractor, and exploited as input for training the RF classifier. Tests conducted on data augmentation show that it does not significantly affect overall performances, but still provide proper support for those categories with fewer points. On the other side, the tests on the fine-tuning have given rise to manifold considerations. Firstly, the standard fine-tuning can achieve performances almost equal to those where only the DGCNN is used, considerably improving some categories. Thus, they confirm that, once the DNN is pre-trained, data processing and prediction times can be significantly reduced (from ca. 48 to 0.5 h), in the case of heritage point clouds too. Then, performances similar to the reference tests are obtained also with the use of the DGCNN as a feature extractor and the RF as a classifier, demonstrating that the final classifier does not affect the prediction

Transferencia de técnicas de aprendizaje y mejora del rendimiento en la segmentación semántica profunda de nubes de puntos del patrimonio construido

[EN] The growing availability of three-dimensional (3D) data, such as point clouds, coming from Light Detection and Ranging (LiDAR), Mobile Mapping Systems (MMSs) or Unmanned Aerial Vehicles (UAVs), provides the opportunity to rapidly generate 3D models to support the restoration, conservation, and safeguarding activities of cultural heritage (CH). The so-called scan-to-BIM process can, in fact, benefit from such data, and they can themselves be a source for further analyses or activities on the archaeological and built heritage. There are several ways to exploit this type of data, such as Historic Building Information Modelling (HBIM), mesh creation, rasterisation, classification, and semantic segmentation. The latter, referring to point clouds, is a trending topic not only in the CH domain but also in other fields like autonomous navigation, medicine or retail. Precisely in these sectors, the task of semantic segmentation has been mainly exploited and developed with artificial intelligence techniques. In particular, machine learning (ML) algorithms, and their deep learning (DL) subset, are increasingly applied and have established a solid state-of-the-art in the last half-decade. However, applications of DL techniques on heritage point clouds are still scarce; therefore, we propose to tackle this framework within the built heritage field. Starting from some previous tests with the Dynamic Graph Convolutional Neural Network (DGCNN), in this contribution close attention is paid to: i) the investigation of fine-tuned models, used as a transfer learning technique, ii) the combination of external classifiers, such as Random Forest (RF), with the artificial neural network, and iii) the evaluation of the data augmentation results for the domain-specific ArCH dataset. Finally, after taking into account the main advantages and criticalities, considerations are made on the possibility to profit by this methodology also for non-programming or domain experts.[ES] La creciente disponibilidad de datos tridimensionales (3D), como nubes de puntos, provenientes de la detección de la luz y distancia (LiDAR), sistemas de mapeado móvil (MMS) o vehículos aéreos no tripulados (UAV), brinda la oportunidad de generar rápidamente modelos 3D para apoyar las actividades de restauración, conservación y salvaguardia del patrimonio cultural (CH). El llamado proceso de escaneado-a-BIM puede, de hecho, beneficiarse de dichos datos, y ellos mismos pueden ser una fuente para futuros análisis o actividades sobre el patrimonio arqueológico y el construido. Hay varias formas de explotar este tipo de datos, como el modelado de información de edificios históricos (HBIM), la creación de mallas, la rasterización, la clasificación y la segmentación semántica. Este último, referido a las nubes de puntos, es un tema de máxima actualidad no solo en el dominio del PC sino también en otros campos como la navegación autónoma, la medicina o el comercio minorista. Precisamente en estos sectores, la tarea de la segmentación semántica se ha explotado y desarrollado principalmente con técnicas de inteligencia artificial. En particular, los algoritmos de aprendizaje automático (AA) y su subconjunto de aprendizaje profundo (AP) se aplican cada vez más y han establecido un sólido estado de la técnica en la última media década. Sin embargo, las aplicaciones de las técnicas de AP en las nubes de puntos tradicionales son todavía escasas; por tanto, nos proponemos abordar este marco dentro del ámbito del patrimonio construido. Partiendo de algunas pruebas anteriores con la Red Neural Convolucional de Gráfico Dinámico (DGCNN), en esta contribución se presta atención a: i) la investigación de modelos afinados, utilizados como técnica de aprendizaje por transferencia, ii) la combinación de clasificadores externos, como Random Forest (RF), con la red neuronal artificial, y iii) la evaluación de los resultados de aumentación de datos para el conjunto de datos específico del dominio ArCH. Finalmente, después de tener en cuenta las principales ventajas y criticidades, se hace una consideración sobre la posibilidad de beneficiarse de esta metodología también a expertos no programadores o del campo.Matrone, F.; Martini, M. (2021). Transfer learning and performance enhancement techniques for deep semantic segmentation of built heritage point clouds. Virtual Archaeology Review. 12(25):73-84. https://doi.org/10.4995/var.2021.15318OJS73841225Armeni, I., Sener, O., Zamir, A. R., Jiang, H., Brilakis, I., Fischer, M., & Savarese, S. (2016). 3D semantic parsing of large-scale indoor spaces. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 1534-1543. https://doi.org/10.1109/CVPR.2016.170Baraldi, L., Cornia, M., Grana, C., & Cucchiara, R. (2018). Aligning text and document illustrations: towards visually explainable digital humanities. In 24th International Conference on Pattern Recognition (ICPR), 1097-1102. IEEE. https://doi.org/10.1109/ICPR.2018.8545064Bassier, M., Yousefzadeh, M., & Vergauwen, M. (2020). Comparison of 2D and 3D wall reconstruction algorithms from point cloud data for as-built BIM. Journal of Information Technology in Construction (ITcon), 25(11), 173-192. https://doi.org/10.36680/j.itcon.2020.011Boulch, A., Guerry, J., Le Saux, B., & Audebert, N. (2018). SnapNet: 3D point cloud semantic labeling with 2D deep segmentation networks. Computers & Graphics, 71, 189-198. https://doi.org/10.1016/j.cag.2017.11.010Chadwick, J., (2020). Google launches hieroglyphics translator that uses AI to decipher images of Ancient Egyptian script. Available at https://www.dailymail.co.uk/sciencetech/article-8540329/Google-launches-hieroglyphics-translator-uses-AI-decipher-Ancient-Egyptian-script.html Last access 24/11/2020Fiorucci, M., Khoroshiltseva, M., Pontil, M., Traviglia, A., Del Bue, A., & James, S. (2020). Machine learning for cultural heritage: a survey. Pattern Recognition Letters, 133, 102-108. https://doi.org/10.1016/j.patrec.2020.02.017Geiger, A., Lenz, P., Stiller, C., & Urtasun, R. (2013). Vision meets robotics: The KITTI dataset. The International Journal of Robotics Research, 32(11), 1231-1237. https://doi.org/10.1177/0278364913491297Grilli, E., & Remondino, F. (2019). Classification of 3D digital heritage. Remote Sensing, 11(7), 847. https://doi.org/10.3390/rs11070847Grilli, E., & Remondino, F. (2020). Machine learning generalisation across different 3D architectural heritage. ISPRS International Journal of Geo-Information, 9(6), 379. https://doi.org/10.3390/ijgi9060379Grilli, E., Özdemir, E., & Remondino, F. (2019a). Application Of Machine And Deep Learning Strategies For The Classification Of Heritage Point Clouds. International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, XLII-4/W18, 447-454, 2019. https://doi.org/10.5194/isprs-archives-XLII-4-W18-447-2019Grilli, E., Farella, E. M., Torresani, A., & Remondino, F. (2019b). Geometric features analysis for the classification of cultural heritage point clouds. International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, XLII-2/W15, 541-548, 2019 https://doi.org/10.5194/isprs-archives-XLII-2-W15-541-2019Hackel, T., Savinov, N., Ladicky, L., Wegner, J. D., Schindler, K., & Pollefeys, M. (2017). Semantic3d.net: A new large-scale point cloud classification benchmark. arXiv:1704.03847He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, 770-778. arXiv:1512.03385Korc, F., & Förstner, W. (2009). eTRIMS Image Database for interpreting images of man-made scenes. Dept. of Photogrammetry, University of Bonn, Tech. Rep. TR-IGG-P-2009-01.Landrieu, L., & Simonovsky, M. (2018). Large-scale point cloud semantic segmentation with superpoint graphs. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 4558-4567. arXiv:1711.09869Llamas, J., M Lerones, P., Medina, R., Zalama, E., & Gómez-García-Bermejo, J. (2017). Classification of architectural heritage images using deep learning techniques. Applied Sciences, 7(10), 992. https://doi.org/10.3390/app7100992Mathias, M., Martinovic, A., Weissenberg, J., Haegler, S., & VanGool, L. (2011). Automatic architectural style recognition. International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, XXXVIII-5/W16, 171-176 3. https://doi.org/10.3390/app7100992Matrone, F., Grilli, E., Martini, M., Paolanti, M., Pierdicca, R., & Remondino, F. (2020a). Comparing machine and deep learning methods for large 3D heritage semantic segmentation. ISPRS International Journal of Geo-Information, 9(9), 535. https://doi.org/10.3390/ijgi9090535Matrone, F., Lingua, A., Pierdicca, R., Malinverni, E. S., Paolanti, M., Grilli, E., Remondino, F., Murtiyoso, A., & Landes, T. (2020b). A benchmark for large-scale heritage point cloud semantic segmentation. International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, XLIII-B2-2020, 1419-1426. https://doi.org/10.5194/isprs-archives-XLIII-B2-2020-1419-2020Murtiyoso, A., & Grussenmeyer, P. (2019a). Automatic heritage building point cloud segmentation and classification using geometrical rules. International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, XLII-2/W15, 821-827. https://doi.org/10.5194/isprs-archives-XLII-2-W15-821-2019Murtiyoso, A., & Grussenmeyer, P. (2019b). Point cloud segmentation and semantic annotation aided by GIS data for heritage complexes. International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, XLII-2/W9, 523-528, 2019. https://doi.org/10.5194/isprs-archives-XLII-2-W9-523-2019Oses, N., Dornaika, F., & Moujahid, A. (2014). Image-based delineation and classification of built heritage masonry. Remote Sensing, 6(3), 1863-1889. https://doi.org/10.3390/rs6031863Park, Y., & Guldmann, J. M. (2019). Creating 3D city models with building footprints and LIDAR point cloud classification: A machine learning approach. Computers, Environment and Urban Systems, 75, 76-89. https://doi.org/10.1016/j.compenvurbsys.2019.01.004Pierdicca, R., Paolanti, M., Matrone, F., Martini, M., Morbidoni, C., Malinverni, E. S. & Lingua, A. M. (2020). Point cloud semantic segmentation using a deep learning framework for cultural heritage. Remote Sensing, 12(6), 1005. https://doi.org/10.3390/rs12061005Qi, C. R., Su, H., Mo, K., & Guibas, L. J. (2017). Pointnet: Deep learning on point sets for 3D classification and segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, 652-660. arXiv:1612.00593Sharafi, S., Fouladvand, S., Simpson, I., & Alvarez, J. A. B. (2016). Application of pattern recognition in detection of buried archaeological sites based on analysing environmental variables, Khorramabad Plain, West Iran. Journal of Archaeological Science: Reports, 8, 206-215. https://doi.org/10.1016/j.jasrep.2016.06.024Simonyan, K., & Zisserman, A. (2014). Very deep convolutional networks for large-scale image recognition. arXiv:1409.1556Stathopoulou, E. K., & Remondino, F. (2019). Semantic photogrammetry: boosting image-based 3D reconstruction with semantic labeling. International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, 42(2), W9. https://doi.org/10.5194/isprs-archives-XLII-2-W9-685-2019Teboul, O., Kokkinos, I., Simon, L., Koutsourakis, P., & Paragios, N. (2012). Parsing facades with shape grammars and reinforcement learning. IEEE transactions on pattern analysis and machine intelligence, 35(7), 1744-1756. https://doi.org/10.1109/TPAMI.2012.252.Teruggi, S., Grilli, E., Russo, M., Fassi, F., & Remondino, F. (2020). A hierarchical machine learning approach for multi-level and multi-resolution 3D point cloud classification. Remote Sensing, 12(16), 2598. https://doi.org/10.3390/rs12162598Tyleček, R., & Šára, R. (2013). Spatial pattern templates for recognition of objects with regular structure. In German Conference on Pattern Recognition, Springer, Berlin, Heidelberg, 364-374. https://doi.org/10.1007/978-3-642-40602-7_39Verschoof-van der Vaart, W. B., & Lambers, K. (2019). Learning to Look at LiDAR: the use of R-CNN in the automated detection of archaeological objects in LiDAR data from the Netherlands. Journal of Computer Applications in Archaeology, 2(1). https://doi.org/10.5334/jcaa.32Wang, Y., Sun, Y., Liu, Z., Sarma, S. E., Bronstein, M. M., &Solomon, J. M. (2019). Dynamic graph CNN for learning on point clouds. ACM Transactions On Graphics, 38(5), 1-12. arXiv:1801.07829Weinmann, M., Jutzi, B., Hinz, S., & Mallet, C. (2015). Semantic point cloud interpretation based on optimal neighborhoods, relevant features and efficient classifiers. ISPRS Journal of Photogrammetry and Remote Sensing, 105, 286-304. https://doi.org/10.1016/j.isprsjprs.2015.01.016Xie, Y., Tian, J., & Zhu, X. X. (2019). Linking points with labels in 3D: a review of point cloud semantic segmentation. arXiv:1908.08854Yan, H., Ding, Y., Li, P., Wang, Q., Xu, Y., & Zuo, W. (2017). Mind the class weight bias: Weighted maximum mean discrepancy for unsupervised domain adaptation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, (pp. 2272-2281). arXiv:1705.00609 https://doi.org/10.1109/CVPR.2017.10

Planned maintenance for architectural heritage. Experiences in progress from 3D survey to intervention programmes through HBIM

The continuous evolution of 3D surveying and modelling techniques, using increasingly high-performance tools and applications, highlights the added value of these methods in the field of urban and architectural survey. In the case study presented, attention is focused particularly on applications useful for the planned maintenance of cultural heritage (CH). These tools and methods have a significant impact on the phase of interpretation and "physical" knowledge. They can also bring a critical contribution to the completion of models that are not only geometric but also semantic and informative, supporting 360-degree planning of the maintenance of our historical architectural heritage. This support for scheduled maintenance has been identified in the HBIM methodology, based on an integrated 3D metric survey. A three-year research project on this topic (Interreg Italy-Switzerland "MAIN.10.ANCE", 2019-2022), partnered by Politecnico di Torino, is currently in the start-up phase. The main focus of the project is a UNESCO heritage site: the "Sacri Monti" (Sacred Mountains) of Italy and those in Canton of Ticino (Switzerland), with the need for a common and shared conservation plan

From scan-to-BIM to a structural finite elements model of built heritage for dynamic simulation

The progress in information technology allows an innovative transformation of practices commonly involved in the engineering and construction field, especially in relation to the existing architectural heritage’s control and management activities. The proposed methodology takes advantage of an integrated 3D metric survey as a basis for an HBIM (Historic Building Information Modelling) model to be exploited for the definition of a Finite Elements Model (FEM). This paper aims to show the applicability of a digital process, stemmed from the integration in Rhinoceros 3D of a BIM structural model, leading to the dynamic simulation of the analytical FEM through PRO_SAP® (a PROfessional Structural Analysis Program). The described workflow investigates the interoperability issues, along with the difficulties in the Scan-to-HBIM processes, demonstrating how HBIM models can anyhow support operations aimed at maintaining and preserving existing historical assets, also from a structural point of view, even if with still persistent criticalities

Il canale Cavour e le risaie: iconografia del paesaggio risicolo piemontese in trasformazione.

The research field is the landscape shaped from the early 19th century in the Piedmont plain among Vercelli, Novara and the Lomellina region after the Cavour canal construction (a hydraulic engineering work realized from the 1863 till the 1866 to mainly support the rice cultivation) whose effects have deeply characterized the agricultural landscape provoking cultural, social and architectural phenomena which became later a local landmark. Through the comparison between the cartography of the pre or post Cavour canal construction (from the teresian cadastre to the 21st century maps), the iconography, such us Gazzone or Ravello’s paintings, the first pictures from the Touring Italian Club Archive, or even some movies, as Riso Amaro and La Risaia, and the latest documentaries, the peculiar elements of this landscape have been identified in order to better understand this territory and be aware of which context features preserve in case of safeguard or valorization of this heritage

Il database spaziale e la sua gestione.

La digitalizzazione nel settore delle costruzioni sta offrendo opportunità significative per l’intera filiera delle costruzioni, migliorando le pratiche esistenti, integrando tecnologie e strumenti dirompenti che possono portare a nuovi processi, modelli di business, materiali e soluzioni, con significative potenzialità anche per la gestione del patrimonio architettonico e per le sfide rappresentate dagli obiettivi di riqualificazione del patrimonio esistente. La corretta combinazione di Digitization, Digitalization e Digital trasformation offre l’opportunità di raccogliere le sfide di quest’epoca definendo nuovi metodi e strumenti di lavoro per innovare l’industria delle costruzioni. La digitalizzazione del settore delle costruzioni si concentra su una serie di tematiche legate a tre categorie principali: i) tecnologie di acquisizione dei dati (e.g. sensori); ii) processi di automazione (e.g. robotica); informazioni e analisi digitali (e.g. Building Information Modelling - BIM) per la rappresentazione grafica

3D model generation using oblique images acquired by UAV

In recent years, many studies revealed the advantages of using airborne oblique images for obtaining improved 3D city models (including façades and building footprints). Here the acquisition and use of oblique images from a low cost and open source Unmanned Aerial Vehicle (UAV) for the 3D high-level-of-detail reconstruction of historical architectures is evaluated. The critical issues of such acquisitions (flight planning strategies, ground control points distribution, etc.) are described. Several problems should be considered in the flight planning: best approach to cover the whole object with the minimum time of flight; visibility of vertical structures; occlusions due to the context; acquisition of all the parts of the objects (the closest and the farthest) with similar resolution; suitable camera inclination, and so on. In this paper a solution is proposed in order to acquire oblique images with one only flight. The data processing was realized using Structure-from-Motion-based approach for point cloud generation using dense image-matching algorithms implemented in an open source software. The achieved results are analysed considering some check points and some reference LiDAR data. The system was tested for surveying a historical architectonical complex: the “Sacro Monte di Varallo Sesia” in north-west of Italy. This study demonstrates that the use of oblique images acquired from a low cost UAV system and processed through an open source software is an effective methodology to survey cultural heritage, characterized by limited accessibility, need for detail and rapidity of the acquisition phase, and often reduced budgets

A BIM-GIS Integrated Database to Support Planned Maintenance Activities of Historical Built Heritage.

Planned maintenance represents a strategy to facilitate the conservation of architectural heritage, preventing invasive restoration activities. For this purpose, the management of a maintenance plan through the integration of BIM and GIS domains is here proposed. In particular, the first results of the Interreg Main.10.ance project are described, namely the definition of a unique spatial database divided into different Levels of Detail, compliant with geographical standards and user-friendly for the professionals involved. This integration is addressed through the use of Dynamo, which allows the dialogue between the BIM and GIS data in the PostgreSQL databas

Detection of Wet Riparian Areas using Very High Resolution Multispectral UAS Imagery Based on a Feature-based Machine Learning Algorithm

Unmanned Aerial System (UAS) imagery has enabled very high-resolution multispectral image acquisition. Detection of wet areas and classification of land cover based on these images using the Machine Learning (ML) algorithm named Random Forest (RF) is our main purpose in this paper. Very high-resolution UAS images have been used as inputs for a machine learner to access the capability of different spectral bands and spectral vegetation indices, elevation, and texture features in the classification of land cover and detection of the wet riparian area in the case study in two different epochs. There are many existing methods for the classification of land cover based on UAS images, but very high-resolution centimeter-level data are of main importance in this analysis. Outstanding results have been produced in both epochs considering three extremely accurate performance analysers. Additionally, in this research, the most decisive and effective features have been discovered to compromise accuracy and the number of effectual features